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Final  Report 

Wavelets  and  neural  networks 
AFOSR  grant  number:  F49620-97-1-0211 
April  1,  1997  -  December  31,  1999 


1.  Objectives 

The  research  supported  by  this  grant  is  a  continuation  of  our  research, 
supported  by  AFOSR  grant  F49620-93-1-0150  (February  15,  1993-  July  15, 
1996),  regarding  the  approximation  properties  of  neural  networks.  We  had 
noticed  several  similarities  in  the  theoretical  aspects  of  wavelets  and  neural 
networks.  The  objective  of  the  current  project  is  to  investigate  these  simi¬ 
larities  in  further  detail. 

2.  Background 

During  our  previous  work,  we  had  established  a  very  close  connection 
between  polynomial  approximation  and  approximation  by  neural  networks. 
In  fact,  we  had  developed  a  unified  theory  of  the  approximation  properties 
of  neural  networks,  radial  basis  function  (RBF)  networks,  and  generalized 
regularization  networks.  Our  networks  provided  an  optimal  approximation 
to  a  class  of  functions,  where  the  only  known  a  priori  assumption  was  the 
number  of  continuous  derivatives.  The  networks  did  not  require  any  training 
in  the  classical  sense,  but  were  given  explicitly  in  terms  of  the  coefficients  of 
the  target  function  in  certain  orthogonal  expansions.  Our  current  objectives 
are  the  following. 

•  Modify  the  formulas  for  the  networks,  so  that  the  networks  can  be 
obtained  in  terms  of  the  values  of  the  target  function  at  judiciously 
chosen  points. 

•  Develop  polynomial  wavelets  with  an  eventual  objective  of  integrating 
these  with  the  theory  of  “generalized  translation  networks”  which  we 
had  previously  developed. 

3.  Accomplishments 

The  research  on  this  project  resulted  in  13  publications.  In  addition,  I 
gave  23  invited  lectures  in  many  different  countries. 
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In  order  to  accomplish  the  objective  (a)  above,  we  introduced  in  [l]  cer¬ 
tain  polynomial  operators,  similar  to  the  well  known  de  la  Vallee  Poussin 
operators  in  the  theory  of  trigonometric  approximation.  In  addition  to  the 
properties  of  the  classical  operators,  these  have  the  added  advantage  that 
their  computation  involves  only  the  values  of  the  function  at  the  nodes  of 
classical  orthogonal  polynomials,  in  particular,  the  Jacobi  and  Freud  poly¬ 
nomials.  In  [1]  and  [2],  we  studied  the  boundedness  of  these  operators  in 
various  norms.  An  interesting  application  of  these  bounds  is  that  the  supre- 
mum  norm  of  a  polynomial  over  a  continuous  interval  is  shown  to  be  of  the 
same  order  of  magnitude  as  the  maximum  absolute  value  of  this  polynomial 
at  these  zeros. 

As  an  application  of  this  theory  and  our  previous  work,  we  demonstrated 
in  [3]  how  generalized  translation  networks  can  be  developed  to  yield  an  op¬ 
timal  approximation  to  functions  in  the  Sobolev  classes,  knowing  only  the 
values  of  the  function  at  nodes  obtained  from  the  zeros  of  orthogonal  poly¬ 
nomials.  From  a  different  point  of  view,  these  nodes  provide  the  “correct” 
choice  of  nodes  for  universal  approximation  of  a  Sobolev  class  in  active  learn¬ 
ing  environment. 

In  [4],  we  report  on  our  experiments  on  the  calibration  of  a  degraded 
phased  array  antenna  using  my  previous  ideas  on  the  theory  of  neural  net¬ 
works  for  function  approximation.  In  many  situations,  the  experiments  re¬ 
flect  a  50%  to  100%  improvement  on  prior  work  conducted  by  Southall  et. 
al.  at  Hanscom  Air  Force  Base. 

Next,  we  reformulated  the  problem  of  multiple  source  direction  finding 
using  phased  array  antennas  as  the  problem  of  finding  the  lo  cation  of  the 
discontinuities  of  certain  derivatives  (singularities  of  different  orders)  of  a 
function,  given  its  coefficients  in  the  Fourier  or  Chebyshev  polynomial  ex¬ 
pansion.  The  classical,  compactly  supported  wavelets  are  not  suitable  for 
this  task,  partly  because  their  construction  requires  the  knowledge  of  the 
corresponding  wavelet  coefficients,  and  partly  because  of  inherent  theoretical 
limitations. 

In  [5] ,  we  gave  a  very  general  construction  of  a  large  class  of  such  polyno¬ 
mial  frames  which  utilize  directly  the  Chebyshev  coefficients,  and  are  free  of 
the  above  limitations.  Our  frames  can  be  defined  using  an  arbitrary  system 
of  orthogonal  polynomials.  The  frame  bounds  in  the  least  squares  sense  are 
independent  of  the  system  of  orthogonal  polynomials  used.  In  the  case  of 
Jacobi  polynomials  (Chebyshev  polynomials  in  particular),  our  results  in  [1], 
[2]  are  used  to  establish  frame  bounds  also  in  the  uniform  and  other  norms. 
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The  available  multiplier  theorems  do  not  apply  in  this  case;  new  ideas  were 
essential.  Our  polynomial  frames  are  able  to  simultaneously  detect  singu¬ 
larities  of  different  orders.  We  have  given  a  precise  quantitative  description 
connecting  the  “largeness”  of  the  frame  coefficients  and  the  order  and  lo¬ 
cation  of  the  singularities.  Applied  to  the  multiple  source  direction  finding 
problem,  our  methods  appear  to  be  remarkably  stable  under  noise. 

In  [6],  we  used  complex  analytic  techniques  to  obtain  yet  another  poly¬ 
nomial  operator  for  the  detection  of  singularities  in  a  function,  given  its 
Chebyshev  coefficients.  Although  these  operators  are  not  frame  operators, 
they  have  much  sharper  localization  properties;  their  values  decay  exponen¬ 
tially  rapidly  as  we  move  away  from  the  singularities. 

During  our  research  on  polynomial  frames,  we  became  aware  of  several 
other  applications  of  the  theme  which  we  were  developing.  In  [7],  we  carried 
out  a  very  general  study  of  periodic  convolution  transforms  for  the  detection 
of  singularities  of  a  periodic  function,  knowing  finitely  many  of  its  Fourier 
coefficients.  Our  theory  enables  us  to  compare  a  variety  of  known  periodic 
wavelets,  as  well  as  to  construct  new  frames  and  wavelets  with  different 
desired  properties.  We  also  constructed  “build-up”  frames,  based  on  random 
samples  of  the  target  functions.  In  classical  wavelet  theory,  one  starts  with  a 
large  number  of  data,  and  compresses  this  information  into  a  small  number 
of  large  wavelet  coefficients.  Our  objective  in  the  construction  of  the  build¬ 
up  frames  is  to  start  with  a  small  number  of  samples,  and  then  enlarge  this 
data  as  needed  to  build  higher  and  higher  degree  frames,  until  the  detection 
of  singularities  is  accomplished  with  a  desired  accuracy.  As  before,  the  same 
theory  can  also  be  used  to  modify  our  earlier  construction  of  neural  networks 
so  as  to  utilize  incremental  amounts  of  random  samples  to  increase  the  size 
of  the  network  as  needed  to  achieve  a  desired  order  of  approximation. 

In.  [8],  we  constructed  build-up  frames  based  on  a  discrete  orthogonal  poly¬ 
nomial  system.  This  construction  was  motivated  by  applications  in  quantum 
mechanics,  pointed  out  to  us  by  a  Russian  physicist. 

We  next  turned  our  attention  to  the  approximation  of  zonal  function 
networks  on  the  sphere.  Our  objective  was  to  give  results  analogous  to  our 
constructions  of  neural  networks,  but  the  form  of  the  neural  networks  is 
different  for  the  sphere:  one  does  not  allow  thresholds,  and  the  weights  are 
limited  to  the  sphere.  In  [10],  we  constructed  zonal  function  networks  that 
provide  an  optimal  order  of  approximation  to  a  large  class  of  function  spaces 
defined  in  terms  of  the  Beltrami  operator  on  the  sphere.  We  have  given 
explicit  formulas  to  “train”  these  networks  using  scattered  data  on  the  sphere 
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in  a  coordinate- free  manner. 

The  paper  [9]  develops  certain  important  and  technical  quadrature  for¬ 
mulas  based  on  scattered  data  on  the  sphere.  These  quadrature  formulas 
play  a  crucial  role  in  [10]  and  [7]. 

The  paper  [11]  is  an  invited  survey  paper  on  our  work  on  neural  networks. 
A  more  extended  survey  is  given  in  a  five  lecture  invited  tutorial  we  gave 
during  an  international  conference  on  wavelets  and  related  topics  in  Delhi, 
India  [13].  The  paper  [12]  surveys  our  work  and  other  related  work  on  the 
zonal  function  network  approximation  on  the  sphere. 

4.  Personnel  supported 

Hrushikesh  N.  Mhaskar,  PI. 

5.  Technical  Publications 


1.  Bounded  quasi-interpolatory  polynomial  operators,  Journal  of  Approx¬ 
imation  Theory,  96  (1999),  67-85.  (With  J.  Prestin). 

2.  On  Marcinkiewicz-Zygmund-type  inequalities,  in  “Approximation  the¬ 
ory:  in  memory  of  A.  K.  Varma”,  (N.  K.  Govil,  R.  N.  Mohapatra, 
Z.  Nashed,  A.  Sharma,  and  J.  Szabados  Eds.),  Marcel  Dekker,  1998, 
pp. 389-404.  (With  J.  Prestin). 

3.  On  a  choice  of  sampling  nodes  for  optimal  approximation  of  smooth 
functions  by  generalized  translation  networks,  in  “Artificial  Neural  Net¬ 
works,  Conference  Publication  No.  440”  (IEE),  1997,  pp.  210-215. 
(With  J.  Prestin). 

4.  Neural  beam-steering  and  direction  finding,  in  “Neural  Networks  in  En¬ 
gineering  Systems”,  (A.  B.  Bulsari  and  S.  Kallio  Eds.),  Royal  Institute 
of  Technology,  Stockholm,  1997,  pp.  269-272.  (With  H.  Southall). 

5.  Polynomial  frames  for  the  detection  of  singularities,  in  “Wavelet  Analy¬ 
sis  and  Multiresolution  Methods”  (Ed.  Tian-Xiao  He),  Lecture  Notes 
in  Pure  and  Applied  Mathematics,  Vol.  212,  Marcel  Decker,  2000, 
273-298.  (With  J.  Prestin). 

6.  On  a  sequence  of  fast  decreasing  polynomial  operators,  in  “Applications 
and  Computation  of  Orthogonal  Polynomials”  (Eds.  W.  Gautschi, 
G.H.  Golub,  G.  Opfer)  Internat.  Ser.  Numer.  Math.,  Birkhauser, 
Basel,  1999,  165-178.  (With  J.  Prestin). 
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7.  On  the  detection  of  singularities  of  a  periodic  function.  Advances  in 
Computational  Mathematics,  12  (2000),  95-131  (With  J.  Prestin). 

8.  On  a  build-up  polynomial  frame  for  the  detection  of  singularities,  in 
“Self-Similar  Systems”  (V.  B.  Priezzhev  and  V.  P.  Spiridonov  Eds.), 
Joint  Institute  for  Nuclear  Research,  Dubna,  Russia,  1999,  pp.  98-109. 
(With  J.  Prestin). 

9.  Quadrature  Formulas  on  Spheres  Using  Scattered  Data,  To  appear  in 
Mathematics  of  Computation.  (With  F.  J.  Narcowich  and  J.  D.  Ward). 

10.  Approximation  Properties  of  Zonal  Function  Networks  Using  Scattered 
Data  on  the  Sphere,  Advances  in  Computational  Mathematics,  11  (1999), 
121-137  (With  F.  J.  Narcowich  and  J.  D.  Ward). 

11.  Approximation  of  smooth  functions  by  neural  networks,  in  “Dealing 
with  complexity:  A  neural  network  approach”,  (K.  Warwick  et.  al. 
eds),  “Perspectives  in  Neural  Computing”,  Springer  Verlag,  London, 
1998,  pp.  189-204. 

12.  Representing  and  analyzing  scattered  data  on  the  sphere,  To  appear  in 
“Multivariate  approximation  and  applications”  (A.  Pinkus,  D.  Levi- 
atan,  N.  Dyn,  and  D.  Levin  Eds.),  Cambridge  University  Press,  Cam¬ 
bridge.  (With  F.  J.  Narcowich  and  J.  D.  Ward). 

13.  Approximation  theory  and  neural  networks,  accepted  for  publication  in 
’’Wavelet  Analysis  and  Applications,  Proceedings  of  the  international 
workshop  in  Delhi,  1999”  (P.  K.  Jain,  M.  Krishnan,  H.  N.  Mhaskar  J. 
Prestin,  and  D.  Singh  Eds.),  Narosa  Publishing,  New  Delhi,  India. 


6.  Interactions 

I  gave  the  following  invited  lectures  as  indicated. 

1.  Ohio  State  University,  Columbus,  Ohio,  April  1997. 

2.  Oak  Ridge  National  Laboratory,  Oak  Ridge,  Tennessee,  April  1997. 

3.  University  of  Kentucky,  Lexington,  Kentucky,  April  1997. 

4.  Technishe  Universitatt,  Munich,  Germany,  July  1977. 

5.  Katholische  Universitatt,  Eichstaett,  Germany,  July  1977. 

6.  International  symposium  on  Approximation  Theory,  Nashville,  Ten¬ 
nessee,  January,  1998.  (Also  organized  and  chaired  a  special  session  on 
neural  networks.) 
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7.  Oberwolfach,  Germany,  March,  1998. 

8.  Montecateni,  Italy,  June,  1998. 

9.  Eichstaett,  Germany,  June,  1998. 

1G.  Dortmund,  Germany,  July,  1998. 

11.  Hohenheim,  Germany,  July,  1998. 

12.  Goettingen,  Germany,  July,  1998. 

13.  Ukrainian  Academy  of  Sciences,  Kiev,  Ukraine,  July,  1998. 

14.  International  Symposium  on  Self-similar  systems,  Dubna,  Russia,  Au¬ 
gust,  1998. 

15.  International  Conference  on  signal  conference,  Eilat,  Israel,  September, 

1998. 

16.  Special  session  on  Wavelet  Analysis,  American  Mathematical  Society, 
Urbana-Champaign,  Illinois,  March,  1999. 

17.  Air  Force  Institute  of  Technology,  Dayton,  Ohio,  March,  1999. 

18.  International  Conference  honoring  R.  S.  Varga,  Kent,  Ohio,  March, 

1999. 

19.  International  Workshop  on  Orthogonal  Polynomials,  Ballenstaedt,  Ger¬ 
many,  April,  1999. 

20.  International  Conference  on  Approximation  Theory,  Kiev,  Ukraine, 
May,  1999. 

21.  International  Conference  on  Computational  Mathematics,  Oxford,  U.K., 
July,  1999. 

22.  Aston  University,  Birmingham,  U.  K.,  July,  1999. 

23.  International  Workshop  on  Wavelet  Analysis  and  related  topics,  Delhi, 
India,  August,  1999. 
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Fellowship,  Alexander  von  Humboldt  Foundation,  May-July  1998. 
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